Vehicles, including autonomous vehicles, can receive data based on the state of the environment around the vehicle including the state of objects in the environment. This data can be used by the autonomous vehicle to perform various functions related to the movement of those objects through the environment. Further, as the vehicle travels through the environment the set of objects in the environment and the state of those objects can also change. As such, the safe operation of an autonomous vehicle in the environment relies on an accurate determination of the state of the environment. Accordingly, there exists a need for a computing system that more effectively determines the state of objects in an environment.